Skip to content

Replace JSON.NET with System.Text.Json across the codebase#2135

Open
niemyjski wants to merge 154 commits into
mainfrom
feature/system-text-json-v2
Open

Replace JSON.NET with System.Text.Json across the codebase#2135
niemyjski wants to merge 154 commits into
mainfrom
feature/system-text-json-v2

Conversation

@niemyjski
Copy link
Copy Markdown
Member

@niemyjski niemyjski commented Feb 28, 2026

Replace JSON.NET with System.Text.Json

Replaces all Newtonsoft.Json serialization with System.Text.Json (STJ) and migrates from NEST to Elastic.Clients.Elasticsearch.

Key changes

Serialization

  • All JSON serialization now uses STJ with SnakeCaseLower naming policy
  • ObjectToInferredTypesConverter handles object-typed properties (replaces DataObjectConverter)
  • [JsonExtensionData] on Event merges root-level @error, @request, etc. into Data dict
  • [JsonPropertyName] overrides for legacy property names (o_s_name, o_s_version)
  • V1 webhook models use [JsonPropertyName] for PascalCase backward compatibility

Elasticsearch

  • Migrated from NEST to Elastic.Clients.Elasticsearch throughout
  • All index configurations use new fluent API
  • Migrations and jobs updated for new ES client
  • Repository queries migrated to Foundatio expression-based API

Event processing

  • All event upgraders migrated from JObject to JsonObject (System.Text.Json.Nodes)
  • All plugins migrated from Newtonsoft serializer to STJ
  • GetValue<T>() simplified — removed unnecessary TryDeserializeWithFallback (PascalCase data never existed in ES)
  • Error write-back fix in ErrorPlugin / SimpleErrorPlugin after pipeline mutation

Removed

  • DataObjectConverter, ElasticJsonNetSerializer, all Newtonsoft classes
  • Foundatio.JsonNet, NEST.JsonNetSerializer, FluentRest.NewtonsoftJson packages
  • Unnecessary PascalCase fallback machinery

Test results

  • 1549 total, 0 failed, 1547 passed, 2 skipped
  • New: 27 serializer tests, model serialization tests, query migration tests, integration tests

Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread tests/Exceptionless.Tests/Serializer/SerializerTests.cs Fixed
Comment thread tests/Exceptionless.Tests/Serializer/SerializerTests.cs Fixed
Comment thread src/Exceptionless.Core/Jobs/EventPostsJob.cs Fixed
Comment thread src/Exceptionless.Web/Controllers/EventController.cs Fixed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Migrates application-layer JSON usage from Newtonsoft.Json to System.Text.Json (STJ), aligning core models/plugins/tests and adding STJ-based infrastructure for Elasticsearch/NEST while keeping Newtonsoft only as a transitive dependency for the NEST wire protocol.

Changes:

  • Introduces STJ-based IElasticsearchSerializer implementation and STJ type metadata modifiers to match prior Newtonsoft serialization behavior (e.g., omit empty collections).
  • Updates core pipeline/plugins/controllers/jobs to use ITextSerializer/JsonSerializerOptions instead of Newtonsoft abstractions.
  • Refactors tests and fixtures to validate STJ semantics (including semantic JSON comparison).

Reviewed changes

Copilot reviewed 87 out of 87 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/Exceptionless.Tests/Utility/DataBuilder.cs Updates test builder to use ITextSerializer for manual stacking info.
tests/Exceptionless.Tests/Serializer/SerializerTests.cs Reworks serializer tests away from Newtonsoft-specific behavior toward STJ round-trips.
tests/Exceptionless.Tests/Serializer/Models/PersistentEventSerializerTests.cs Uses ITextSerializer for typed data retrieval in PersistentEvent tests.
tests/Exceptionless.Tests/Serializer/Models/DataDictionaryTests.cs Updates GetValue<T> tests to use ITextSerializer pathway.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs Compares serialized JSON via ITextSerializer instead of ToJson().
tests/Exceptionless.Tests/Repositories/ProjectRepositoryTests.cs Updates Slack token access to use serializer-aware accessor.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs Uses serializer-based typed accessors in repository tests.
tests/Exceptionless.Tests/Plugins/WebHookDataTests.cs Switches expected/actual comparisons to semantic JSON equivalence using STJ.
tests/Exceptionless.Tests/Plugins/SummaryDataTests.cs Moves summary comparisons to semantic JSON using STJ.
tests/Exceptionless.Tests/Plugins/GeoTests.cs Updates Geo plugin/tests to pass ITextSerializer.
tests/Exceptionless.Tests/Plugins/EventUpgraderTests.cs Uses JsonNode formatting + semantic compare for upgrader outputs.
tests/Exceptionless.Tests/Plugins/EventParserTests.cs Uses ITextSerializer for round-trip verification; removes Newtonsoft formatting asserts.
tests/Exceptionless.Tests/Pipeline/EventPipelineTests.cs Updates pipeline tests to use serializer-based typed accessors.
tests/Exceptionless.Tests/Mail/MailerTests.cs Updates Mailer construction to accept ITextSerializer.
tests/Exceptionless.Tests/IntegrationTestsBase.cs Replaces FluentRest Newtonsoft serializer with STJ JsonContentSerializer.
tests/Exceptionless.Tests/Exceptionless.Tests.csproj Removes FluentRest.NewtonsoftJson, adds STJ-based FluentRest.
tests/Exceptionless.Tests/Controllers/EventControllerTests.cs Uses ITextSerializer for typed accessors in controller tests.
src/Exceptionless.Web/Exceptionless.Web.csproj Removes NEST.JsonNetSerializer package reference.
src/Exceptionless.Web/Controllers/ProjectController.cs Injects ITextSerializer for Slack token access.
src/Exceptionless.Web/Controllers/EventController.cs Injects ITextSerializer and uses it for event payload byte generation.
src/Exceptionless.Core/Utility/TypeHelper.cs Updates equality handling for STJ JsonElement.
src/Exceptionless.Core/Utility/ExtensibleObject.cs Adds JsonElement handling to generic property retrieval.
src/Exceptionless.Core/Utility/ErrorSignature.cs Switches error signature extraction to serializer-based GetValue<T>.
src/Exceptionless.Core/Services/SlackService.cs Uses serializer-aware Slack token access.
src/Exceptionless.Core/Serialization/ObjectToInferredTypesConverter.cs Refines number inference logic for STJ dynamic object conversion.
src/Exceptionless.Core/Serialization/LowerCaseUnderscorePropertyNamesContractResolver.cs Deletes Newtonsoft contract resolver (obsolete after STJ migration).
src/Exceptionless.Core/Serialization/JsonSerializerOptionsExtensions.cs Defines Exceptionless STJ defaults (snake_case, safe encoder, converters, empty-collection skipping).
src/Exceptionless.Core/Serialization/ExceptionlessNamingStrategy.cs Deletes Newtonsoft naming strategy (replaced by STJ naming policy).
src/Exceptionless.Core/Serialization/EmptyCollectionModifier.cs Adds STJ type info modifier to omit empty collections.
src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Adds STJ IElasticsearchSerializer for NEST with custom converters.
src/Exceptionless.Core/Serialization/ElasticJsonNetSerializer.cs Deletes Newtonsoft-based NEST serializer.
src/Exceptionless.Core/Serialization/ElasticConnectionSettingsAwareContractResolver.cs Deletes Newtonsoft resolver used by old NEST serializer.
src/Exceptionless.Core/Serialization/DynamicTypeContractResolver.cs Deletes Newtonsoft dynamic contract resolver.
src/Exceptionless.Core/Serialization/DataObjectConverter.cs Deletes Newtonsoft-based model/data converter.
src/Exceptionless.Core/Repositories/Configuration/ExceptionlessElasticConfiguration.cs Switches Elasticsearch client wiring to STJ serializer.
src/Exceptionless.Core/Plugins/WebHook/Default/010_VersionOnePlugin.cs Uses serializer-based typed accessors; adds [JsonPropertyName] for v1 webhook compatibility.
src/Exceptionless.Core/Plugins/WebHook/Default/005_SlackPlugin.cs Uses serializer-based typed accessors for Slack webhook creation.
src/Exceptionless.Core/Plugins/Formatting/FormattingPluginBase.cs Converts formatting plugins to depend on ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/99_DefaultFormattingPlugin.cs Updates default formatting/slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/60_LogFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/50_SessionFormattingPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/40_UsageFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/30_NotFoundFormattingPlugin.cs Updates typed accessors + IP retrieval to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/20_ErrorFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/10_SimpleErrorFormattingPlugin.cs Updates typed accessors + slack attachment creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/Formatting/Default/05_ManualStackingFormattingPlugin.cs Updates manual stacking info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventUpgrader/EventUpgraderContext.cs Migrates upgrader DOM from JObject/JArray to JsonObject/JsonArray.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V2_EventUpgrade.cs Rewrites upgrader logic from Newtonsoft DOM to JsonNode DOM.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R850_EventUpgrade.cs Rewrites upgrader logic to JsonObject.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R844_EventUpgrade.cs Rewrites upgrader logic to JsonObject.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/V1R500_EventUpgrade.cs Rewrites upgrader logic to JsonObject and normalizes date string formatting.
src/Exceptionless.Core/Plugins/EventUpgrader/Default/GetVersion.cs Rewrites version detection to JsonNode APIs.
src/Exceptionless.Core/Plugins/EventProcessor/Default/90_RemovePrivateInformationPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/80_AngularPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/70_SessionPlugin.cs Updates typed accessors + session start creation to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/50_GeoPlugin.cs Updates IP extraction to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/45_EnvironmentInfoPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/40_RequestInfoPlugin.cs Updates typed accessors and request exclusion processing to accept ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/30_SimpleErrorPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/20_ErrorPlugin.cs Updates typed accessors + ErrorSignature to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/10_NotFoundPlugin.cs Updates typed accessors to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/0_ThrottleBotsPlugin.cs Updates request-info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventProcessor/Default/03_ManualStackingPlugin.cs Updates manual stacking info access to use ITextSerializer.
src/Exceptionless.Core/Plugins/EventParser/Default/LegacyErrorParserPlugin.cs Migrates legacy parsing to JsonNode + STJ options.
src/Exceptionless.Core/Plugins/EventParser/Default/JsonEventParserPlugin.cs Uses ITextSerializer for parsing event payloads instead of Newtonsoft.
src/Exceptionless.Core/Models/SummaryData.cs Adjusts nullability/required-ness to align with STJ behaviors.
src/Exceptionless.Core/Models/StackSummaryModel.cs Adjusts nullability/required-ness to align with STJ behaviors.
src/Exceptionless.Core/Models/Stack.cs Removes Newtonsoft enum converter attribute; keeps STJ converter.
src/Exceptionless.Core/Models/SlackToken.cs Replaces Newtonsoft [JsonProperty] with STJ [JsonPropertyName]; updates SlackAttachment ctor to use ITextSerializer.
src/Exceptionless.Core/Models/Messaging/ReleaseNotification.cs Removes required modifiers (likely to avoid STJ required-member enforcement).
src/Exceptionless.Core/Models/Event.cs Adds [JsonExtensionData] + IJsonOnDeserialized merge into Data.
src/Exceptionless.Core/Mail/Mailer.cs Switches user-info extraction to serializer-based accessors.
src/Exceptionless.Core/Jobs/WebHooksJob.cs Switches webhook POST payload handling to STJ PostAsJsonAsync + options.
src/Exceptionless.Core/Jobs/EventPostsJob.cs Uses serializer-based event bytes for retries.
src/Exceptionless.Core/Jobs/EventNotificationsJob.cs Updates request-info access to use ITextSerializer.
src/Exceptionless.Core/Jobs/CloseInactiveSessionsJob.cs Updates identity extraction to use ITextSerializer.
src/Exceptionless.Core/Extensions/RequestInfoExtensions.cs Requires serializer for post-data exclusion parsing (STJ).
src/Exceptionless.Core/Extensions/ProjectExtensions.cs Makes Slack token extraction serializer-aware via GetValue<T>.
src/Exceptionless.Core/Extensions/PersistentEventExtensions.cs Updates helpers to accept ITextSerializer for typed accessors.
src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Adds STJ JsonNode helpers used by upgraders/tests (formatting, mutation helpers).
src/Exceptionless.Core/Extensions/JsonExtensions.cs Removes Newtonsoft helpers, retains JSON-type detection helpers.
src/Exceptionless.Core/Extensions/EventExtensions.cs Updates typed accessor APIs to accept ITextSerializer; switches GetBytes to serializer bytes.
src/Exceptionless.Core/Extensions/ErrorExtensions.cs Updates stacking target helper to use serializer-based error access.
src/Exceptionless.Core/Extensions/DataDictionaryExtensions.cs Reworks GetValue<T> around STJ/ITextSerializer, adds case-insensitive JsonElement handling.
src/Exceptionless.Core/Exceptionless.Core.csproj Removes Newtonsoft/JsonNet packages.
src/Exceptionless.Core/Bootstrapper.cs Removes Newtonsoft setup; registers STJ options + SystemTextJsonSerializer in DI.
AGENTS.md Documents the new STJ-only serialization architecture and security posture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 87 out of 87 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Serialization/ElasticSystemTextJsonSerializer.cs Outdated
Comment thread src/Exceptionless.Core/Serialization/ObjectToInferredTypesConverter.cs Outdated
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/DataDictionaryExtensions.cs Fixed
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch from e6a3b35 to c292e13 Compare March 1, 2026 00:52
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/JsonNodeExtensions.cs Fixed
Comment thread src/Exceptionless.Core/Extensions/ProjectExtensions.cs Fixed
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch 2 times, most recently from 4d880ad to 09134ce Compare March 1, 2026 01:19
Copilot AI review requested due to automatic review settings March 22, 2026 00:51
@niemyjski niemyjski force-pushed the feature/system-text-json-v2 branch from 09134ce to 13aa277 Compare March 22, 2026 00:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 86 out of 86 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/Exceptionless.Tests/IntegrationTestsBase.cs Outdated
Comment thread tests/Exceptionless.Tests/Exceptionless.Tests.csproj Outdated
Comment thread src/Exceptionless.Core/Exceptionless.Core.csproj Outdated
Comment thread src/Exceptionless.Core/Utility/TypeHelper.cs Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 117 out of 117 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Repositories/OrganizationRepository.cs
Comment thread src/Exceptionless.Core/Jobs/Elastic/DataMigrationJob.cs Fixed
Copilot AI review requested due to automatic review settings March 22, 2026 19:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/Exceptionless.Core/Repositories/OrganizationRepository.cs
Comment thread src/Exceptionless.Core/Serialization/EmptyCollectionModifier.cs
Comment thread tests/Exceptionless.Tests/IntegrationTestsBase.cs
Comment thread src/Exceptionless.Core/Utility/ExtensibleObject.cs Fixed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 121 out of 121 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@niemyjski niemyjski requested a review from Copilot March 22, 2026 21:14
Comment thread src/Exceptionless.Web/Controllers/SavedViewController.cs Outdated
Comment thread tests/Exceptionless.Tests/TestWithServices.cs Outdated
Comment thread src/Exceptionless.AppHost/Exceptionless.AppHost.csproj Outdated
Comment thread tests/Exceptionless.Tests/Search/PersistentEventQueryValidatorTests.cs Outdated
niemyjski added 2 commits May 23, 2026 09:30
…son-v2

# Conflicts:
#	tests/Exceptionless.Tests/Controllers/StackControllerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/InnerErrorSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/LocationSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/ManualStackingInfoSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/MethodSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/OrganizationSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/SavedViewSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/SubmissionClientSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/TokenSerializerTests.cs
#	tests/Exceptionless.Tests/Serializer/Models/UserDescriptionSerializerTests.cs
The ScopedTermRangeQueryVisitor was a workaround for a specific issue
in Foundatio's query parsing where group node fields were not applied
to term range nodes. The updated Foundatio.Repositories.Elasticsearch
package now handles this behavior correctly, making the custom visitor
redundant.

Also updates GetUserIdentity calls in tests to use ITextSerializer,
aligning with System.Text.Json migration.
Comment thread src/Exceptionless.Core/Repositories/Configuration/Indexes/EventIndex.cs Outdated
- Reorder ConfigureFoundatioRepositoryDefaults/ConfigureExceptionlessDefaults
  calls so app-level WhenWritingNull wins over Foundatio's Never default
- Remove Foundatio's ObjectToInferredTypesConverter to avoid conflict with
  our preferInt64 version
- Fix EventControllerTests.Casing tests that had hardcoded dates past the
  3-day event retention window (use dynamic recent dates instead)
- Remove diagnostic code from ProjectRepositoryTests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread tests/Exceptionless.Tests/Controllers/EventControllerTests.Casing.cs Outdated
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 176 out of 177 changed files in this pull request and generated 1 comment.

Comment thread src/Exceptionless.Core/Serialization/JsonNumberInference.cs
- Use 'is' pattern matching instead of '==' for JsonValueKind comparison
- Merge two backward loops into one that removes JsonStringEnumConverter,
  ObjectToInferredTypesConverter (both Exceptionless and Foundatio variants)
- Restore NOTE comment about ExternalAuthInfo camel case keys

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
niemyjski and others added 7 commits May 23, 2026 14:09
- Revert SavedView.SlugPattern/SlugRegex() (out of scope for STJ PR)
- Revert NewSavedView.cs, UpdateSavedView.cs slug regex to original
- Revert SavedViewController.cs to inline regex
- Revert SavedViewControllerTests.cs formatting-only changes
- Revert openapi.json slug pattern change

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Derive date string from captured DateTimeOffset instead of making a
second independent TimeProvider.GetUtcNow() call that could drift.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the generated Elastic mappings equivalent to origin/main while porting the index configuration to the v8 client APIs.

- Preserve NEST's default float number mapping for usage, usage hour, and billing status fields.

- Preserve the token type byte mapping and the non-indexed request OS copy-to field.

- Use an ordinal comparison when resolving mapped JSON property names.
Tighten STJ object number inference so very large numeric tokens do not throw or truncate when they cannot be represented as Int64.

- Prefer Int32/Int64 where possible, then Decimal, then Double for oversized values.

- Document why application serializer options still set converters even though repository source serialization has its own defaults.

- Cover the oversized-number path with serializer tests.
Update the casing assertions to match the current lower-case underscore contract and remove time-dependent expectations from the event casing tests.

- Assert admin settings serialize base_url, not the old base_u_r_l artifact.

- Use a fixed UTC point in casing tests so expected dates are stable across runs.
@niemyjski
Copy link
Copy Markdown
Member Author

I verified mapping parity by creating fresh test indexes from origin/main, capturing raw Elasticsearch mappings, then repeating the same generated-index capture on this branch. The final raw mapping snapshots compare byte-for-byte identical across events, migrations, organizations, projects, saved views, stacks, tokens, users, and webhooks.

The important compatibility points are:

  • usage and usage_hours numeric fields remain float, matching NEST .Number(...)
  • organization billing_status remains float, matching NEST .Number(...)
  • token type remains byte
  • request OS data keeps index: false

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 176 out of 177 changed files in this pull request and generated 4 comments.

Comment on lines +30 to +35
.Keyword(e => e.PlanId)
.Keyword(e => e.PlanName, k => k.IgnoreAbove(256))
.Date(e => e.SubscribeDate)
.FloatNumber(e => e.BillingStatus)
.DoubleNumber(e => e.BillingPrice)
.Boolean(e => e.IsSuspended)
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this as FloatNumber intentionally for backwards-compatible mapping parity. Although BillingStatus is an enum in the domain model, origin/main uses NEST .Number(...) here and the generated raw Elasticsearch mapping is float. I generated fresh indexes from origin/main and this branch and compared the raw mappings across all indexes; the final snapshots are byte-for-byte identical.

Comment on lines 58 to +66
{
return descriptor
.Object<UsageInfo>(ui => ui.Name(o => o.Usage.First()).Properties(p => p
.Date(fu => fu.Name(i => i.Date))
.Number(fu => fu.Name(i => i.Total))
.Number(fu => fu.Name(i => i.Blocked))
.Number(fu => fu.Name(i => i.Discarded))
.Number(fu => fu.Name(i => i.Limit))
.Number(fu => fu.Name(i => i.TooBig))))
.Object<UsageInfo>(ui => ui.Name(o => o.UsageHours.First()).Properties(p => p
.Date(fu => fu.Name(i => i.Date))
.Number(fu => fu.Name(i => i.Total))
.Number(fu => fu.Name(i => i.Blocked))
.Number(fu => fu.Name(i => i.Discarded))
.Number(fu => fu.Name(i => i.Limit))
.Number(fu => fu.Name(i => i.TooBig))));
.Object(o => o.Usage, ui => ui.Properties(p => p
.Date("date")
.FloatNumber("total")
.FloatNumber("blocked")
.FloatNumber("discarded")
.FloatNumber("limit")
.FloatNumber("too_big")))
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping these usage fields as FloatNumber intentionally for backwards-compatible mapping parity. The domain values are int, but origin/main uses NEST .Number(...), whose generated raw Elasticsearch mapping for these fields is float. I verified this by creating fresh indexes from origin/main, creating fresh indexes from this branch, and comparing the raw mappings across all generated indexes; the final snapshots are byte-for-byte identical.

Comment on lines 49 to +56
return descriptor
.Object<UsageInfo>(ui => ui.Name(o => o.Usage.First()).Properties(p => p
.Date(fu => fu.Name(i => i.Date))
.Number(fu => fu.Name(i => i.Total))
.Number(fu => fu.Name(i => i.Blocked))
.Number(fu => fu.Name(i => i.Discarded))
.Number(fu => fu.Name(i => i.Limit))
.Number(fu => fu.Name(i => i.TooBig))))
.Object<UsageInfo>(ui => ui.Name(o => o.UsageHours.First()).Properties(p => p
.Date(fu => fu.Name(i => i.Date))
.Number(fu => fu.Name(i => i.Total))
.Number(fu => fu.Name(i => i.Blocked))
.Number(fu => fu.Name(i => i.Discarded))
.Number(fu => fu.Name(i => i.Limit))
.Number(fu => fu.Name(i => i.TooBig))));
.Object(o => o.Usage, ui => ui.Properties(p => p
.Date("date")
.FloatNumber("total")
.FloatNumber("blocked")
.FloatNumber("discarded")
.FloatNumber("limit")
.FloatNumber("too_big")))
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the project usage fields as FloatNumber intentionally for backwards-compatible mapping parity. origin/main uses NEST .Number(...) for these fields, and the generated raw Elasticsearch mapping is float. This branch now matches the raw generated main mapping byte-for-byte across the full index set.

Comment on lines +57 to 64
.Object(o => o.UsageHours, ui => ui.Properties(p => p
.Date("date")
.FloatNumber("total")
.FloatNumber("blocked")
.FloatNumber("discarded")
.FloatNumber("limit")
.FloatNumber("too_big")));
}
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same answer here: this is intentionally FloatNumber to preserve the existing generated mapping. Main already maps project usage_hours counters as float via NEST .Number(...). Switching to integer/long would be a schema change and should be handled separately with an index version/reindex plan, not inside the JSON serializer migration.

niemyjski added 2 commits May 23, 2026 16:24
Comprehensive integration tests that capture request.json, elastic.json, and
response.json for all critical API endpoints. Output is written to
dogfood-output/{branch}/ for cross-branch comparison.

Covers: event submission (snake_case, PascalCase, camelCase, mixed, typed
values, legacy SDK format, minimal, feature usage, session), plus
organization/project/token/webhook CRUD endpoints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

fix: add os_name/os_version deserialization aliases for EnvironmentInfo

Older SDKs and snake_case payloads send "os_name" and "os_version" (single
underscore), but the model's [JsonPropertyName] attributes specify "o_s_name"
and "o_s_version" (matching the ES field inferrer's ToLowerUnderscoredWords
output for "OSName"). STJ's PropertyNameCaseInsensitive cannot match these
structurally different strings.

Add private alias properties with [JsonInclude] that accept the alternate form
during deserialization. The aliases use ??= so the canonical form always wins
when both are present. Serialization is unaffected (aliases return null, which
is skipped by DefaultIgnoreCondition.WhenWritingNull).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

test: remove redundant scoped range query coverage

The custom ScopedTermRangeQueryVisitor workaround was removed after updating Foundatio because scoped range query construction is now handled by Foundatio itself.

Keep Exceptionless coverage at the query-rewrite contract level and avoid duplicating Foundatio's low-level query builder behavior here.

Revert "fix: add os_name/os_version deserialization aliases for EnvironmentInfo"

This reverts commit a328df8ccf5c1ad3643bdc379c4f4237cf23d213.

Revert "test: add serialization audit test infrastructure"

This reverts commit fdaee14347fdf748e1af5fc225d7b36a54da92b1.
@github-actions
Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.Web 71% 60% 3753
Exceptionless.AppHost 20% 15% 88
Exceptionless.Insulation 24% 23% 203
Exceptionless.Core 69% 62% 7806
Summary 68% (13340 / 19638) 60% (6892 / 11400) 11850

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants